Session delays by Gallaecio · Pull Request #262 · scrapy-plugins/scrapy-zyte-api

Gallaecio · 2025-08-04T08:36:28Z

See CHANGES.rst for an overview of the changes.

The changes also include a refactoring of session tests, mostly a split of them into ~20 files of less than 500 lines each, extracting stats expectation checks into a helper function, and the merging of a few functions to use parametrize instead. I would not bother reviewing every test file, I recommend focusing on test_sessions_pool, which contains the tests for the main changes.

cc @proway2 @gutsytechster

…a key

codecov · 2026-02-06T21:55:35Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.50%. Comparing base (5b9cd49) to head (178360f).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #262      +/-   ##
==========================================
+ Coverage   97.36%   97.50%   +0.13%     
==========================================
  Files          15       15              
  Lines        1939     2006      +67     
  Branches      356      365       +9     
==========================================
+ Hits         1888     1956      +68     
  Misses         25       25              
+ Partials       26       25       -1

Files with missing lines	Coverage Δ
scrapy_zyte_api/_session.py	`99.60% <100.00%> (+0.29%)`	⬆️

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…e condition

…of a race condition

Copilot

Pull request overview

This PR introduces configurable delays between reuses of plugin-managed sessions (optionally randomized), adds per-pool configuration (including dict-based pool specs), updates docs/terminology, and refactors + expands session test coverage.

Changes:

Add ZYTE_API_SESSION_DELAY + ZYTE_API_SESSION_RANDOMIZE_DELAY, and implement per-session reuse scheduling in the session rotation queue.
Introduce ZYTE_API_SESSION_POOLS (and deprecate ZYTE_API_SESSION_POOL_SIZES), allowing per-pool delay/size/randomize_delay overrides and dict-based pool specs from SessionConfig.pool / request meta.
Split/refactor session tests into many smaller files and add helpers for session stats assertions; update docs and changelog accordingly.

Reviewed changes

Copilot reviewed 32 out of 33 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tox.ini	Pins `typing_extensions` for min environments; updates x402-related deps.
pyproject.toml	Adds runtime dependency on `typing_extensions`; constrains optional `x402`.
scrapy_zyte_api/_session.py	Implements session reuse delays, per-pool configs, improved PoolError messaging, new troubleshooting URL helper.
tests/init.py	Introduces shared `UNSET`, `SESSION_SETTINGS`, and a `get_session_stats` helper.
tests/helpers.py	Adds `assert_session_stats` normalizing helper for stats assertions.
tests/conftest.py	Enables assert rewriting for `tests.helpers`.
tests/test_api_requests.py	Uses shared `UNSET` instead of local definition.
tests/test_addon.py	Adds addon/manual settings parity test for session retry policy settings.
tests/test_request_fingerprinter.py	Updates terminology in a comment.
tests/test_sessions_utils.py	Adds unit test coverage for `is_session_init_request`.
tests/test_sessions_refresh.py	Adds coverage for refresh behavior and concurrent refresh scenarios.
tests/test_sessions_queue.py	Adds coverage for queue empty behavior and max-attempts handling.
tests/test_sessions_pool.py	Adds broad coverage for pool IDs, pool params caching, delays, randomization, and pool sizes/deprecation logging.
tests/test_sessions_poet.py	Adds provider integration coverage with `scrapy_poet`.
tests/test_sessions_max.py	Adds coverage for max bad inits/check failures/errors behavior.
tests/test_sessions_init_url.py	Adds coverage for session init URL precedence (params override vs request URL).
tests/test_sessions_init_precedence.py	Adds exhaustive coverage for params/location precedence across settings + request meta and session configs.
tests/test_sessions_init_location_config.py	Adds coverage for `LocationSessionConfig` location_* hooks and registry reset safety.
tests/test_sessions_init_location.py	Adds coverage for `SessionConfig.location` override behaviors and precedence pitfalls.
tests/test_sessions_init_bad.py	Adds coverage for `ZYTE_API_SESSION_MAX_BAD_INITS_PER_POOL`.
tests/test_sessions_errors.py	Adds coverage for retry stop conditions, missing session IDs, and exception mapping into stats/reasons.
tests/test_sessions_enabled.py	Adds coverage for enabling/disabling sessions via setting/meta and via `SessionConfig.enabled`.
tests/test_sessions_cookies.py	Adds coverage for cookie behavior with/without sessions and `dont_merge_cookies`.
tests/test_sessions_config.py	Adds coverage for `@session_config` behaviors including process_request variants and params errors.
tests/test_sessions_check_errors.py	Adds coverage for checker precedence over max errors.
tests/test_sessions_check_default.py	Adds coverage for default checker’s location-based behavior and unsupported setLocation.
tests/test_sessions_check_custom.py	Adds coverage for custom checker variants, CloseSpider, exceptions, and meta propagation into init requests.
tests/test_sessions_assign.py	Adds coverage for session assignment meta key selection (zyte_api vs automap) and provider consistency.
docs/usage/session.rst	Renames/retunes session terminology; adds troubleshooting section; documents new delay/pooled settings.
docs/reference/settings.rst	Documents new `ZYTE_API_SESSION_DELAY`, `ZYTE_API_SESSION_POOLS`, `ZYTE_API_SESSION_RANDOMIZE_DELAY`; updates wording.
docs/reference/meta.rst	Updates wording for session-enabled and session-pool meta keys (but see suggestion).
CHANGES.rst	Adds 0.33.0 notes covering delays, dict-based pools, deprecation, terminology, troubleshooting section.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

scrapy_zyte_api/_session.py

CHANGES.rst

docs/reference/meta.rst

wRAR · 2026-02-16T13:22:04Z

(this now has conflicts, which probably was expected)

Gallaecio and others added 23 commits August 4, 2025 10:35

WIP

408678e

Merge remote-tracking branch 'origin/main' into session-delays

8d676b8

Change session terminology and add a session troubleshooting section

02c7adf

Further documentation updates

90cd72c

Complete the implementation

af5eb24

Split test_sessions.py into more manageable files

9c2afc9

Use ZYTE_API_SESSION_DELAY=0 by default for tests

05a2635

Update the documentation of the zyte_api_session_pool request metadat…

a25acb9

…a key

Update the ZYTE_API_SESSION_DELAY docs

a3eb5e1

Add test_delay_default

16e92f9

Extend delay tests

e8e66ab

Refactor delay and size tests

c2db031

Improve test coverage

6aeeb50

Test new PoolError scenarios

76b2127

Make ZYTE_API_SESSION_DELAY default to DOWNLOAD_DELAY

8395a62

Implement ZYTE_API_SESSION_RANDOMIZE_DELAY

c01f55b

Consolidate similar tests

667d956

Simplify session stat checking in tests

da2a716

Improve test coverage

77f7d7d

Complete coverage

a46b055

Solve typing issues

aa41ef3

Fix test_missing_session_id_on_response

a4237d1

x402<2.0.0

157a27e

AdrianAtZyte added 4 commits February 6, 2026 23:23

Restore old Scrapy support

60ab006

test_max: see if a non-zero queue time lowers the likelihood of a rac…

257f35d

…e condition

test_empty_queue: see if a non-zero queue time lowers the likelihood …

10d0fdd

…of a race condition

Remove typing_extensions freezing from min-x402

bebde01

Gallaecio requested a review from Copilot February 6, 2026 23:32

Copilot AI reviewed Feb 6, 2026

View reviewed changes

scrapy_zyte_api/_session.py Show resolved Hide resolved

scrapy_zyte_api/_session.py Show resolved Hide resolved

CHANGES.rst Show resolved Hide resolved

CHANGES.rst Outdated Show resolved Hide resolved

docs/reference/meta.rst Outdated Show resolved Hide resolved

AdrianAtZyte added 2 commits February 7, 2026 00:39

Define PoolConfig.randomize_delay

dfa73b4

Address feedback

178360f

Gallaecio marked this pull request as ready for review February 6, 2026 23:49

Gallaecio requested review from kmike and wRAR February 6, 2026 23:49

wRAR approved these changes Feb 18, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Session delays#262

Session delays#262
Gallaecio wants to merge 29 commits intomainfrom
session-delays

Gallaecio commented Aug 4, 2025 •

edited

Loading

Uh oh!

codecov bot commented Feb 6, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wRAR commented Feb 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

Gallaecio commented Aug 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov bot commented Feb 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

wRAR commented Feb 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Gallaecio commented Aug 4, 2025 •

edited

Loading

codecov bot commented Feb 6, 2026 •

edited

Loading